{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## tf.example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "import random\n",
    "import struct\n",
    "import csv\n",
    "from tensorflow.core.example import example_pb2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_bin_files = \"../P007PytorchPointerGeneratorNetwork/cnn-dailymail/finished_files/chunked/train_*\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "filelists = glob.glob(train_bin_files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'../P007PytorchPointerGeneratorNetwork/cnn-dailymail/finished_files/chunked/train_000.bin'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filelists[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "len_bytes: b'\\xd9\\x11\\x00\\x00\\x00\\x00\\x00\\x00'\n",
      "str_len: 4569\n",
      "example_str: b\"\\n\\xd6#\\n\\xdf\\x02\\n\\x08abstract\\x12\\xd2\\x02\\n\\xcf\\x02\\n\\xcc\\x02<s> mentally ill inmates in miami are housed on the `` forgotten floor '' </s> <s> judge steven leifman says most are there as a result of `` avoidable felonies '' </s> <s> while cnn tours facility , patient shouts : `` i am the son of the president '' </s> <s> leifman says the system is unjust and he 's fighting for change . </s>\\n\\xf1 \\n\\x07article\\x12\\xe5 \\n\\xe2 \\n\\xdf editor 's note : in our behind the scenes series , cnn correspondents share their experiences in covering news and analyze the stories behind the events . here , soledad o'brien takes users inside a jail where many of the inmates are mentally ill . an inmate housed on the `` forgotten floor , '' where many mentally ill inmates are housed in miami before trial . miami , florida -lrb- cnn -rrb- -- the ninth floor of the miami-dade pretrial detention facility is dubbed the `` forgotten floor . '' here , inmates with the most severe mental illnesses are incarcerated until they 're ready to appear in court . most often , they face drug charges or charges of assaulting an officer -- charges that judge steven leifman says are usually `` avoidable felonies . '' he says the arrests often result from confrontations with police . mentally ill people often wo n't do what they 're told when police arrive on the scene -- confrontation seems to exacerbate their illness and they become more paranoid , delusional , and less likely to follow directions , according to leifman . so , they end up on the ninth floor severely mentally disturbed , but not getting any real help because they 're in jail . we toured the jail with leifman . he is well known in miami as an advocate for justice and the mentally ill . even though we were not exactly welcomed with open arms by the guards , we were given permission to shoot videotape and tour the floor . go inside the ` forgotten floor ' '' at first , it 's hard to determine where the people are . the prisoners are wearing sleeveless robes . imagine cutting holes for arms and feet in a heavy wool sleeping bag -- that 's kind of what they look like . they 're designed to keep the mentally ill patients from injuring themselves . that 's also why they have no shoes , laces or mattresses . leifman says about one-third of all people in miami-dade county jails are mentally ill . so , he says , the sheer volume is overwhelming the system , and the result is what we see on the ninth floor . of course , it is a jail , so it 's not supposed to be warm and comforting , but the lights glare , the cells are tiny and it 's loud . we see two , sometimes three men -- sometimes in the robes , sometimes naked , lying or sitting in their cells . `` i am the son of the president . you need to get me out of here ! '' one man shouts at me . he is absolutely serious , convinced that help is on the way -- if only he could reach the white house . leifman tells me that these prisoner-patients will often circulate through the system , occasionally stabilizing in a mental hospital , only to return to jail to face their charges . it 's brutally unjust , in his mind , and he has become a strong advocate for changing things in miami . over a meal later , we talk about how things got this way for mental patients . leifman says 200 years ago people were considered `` lunatics '' and they were locked up in jails even if they had no charges against them . they were just considered unfit to be in society . over the years , he says , there was some public outcry , and the mentally ill were moved out of jails and into hospitals . but leifman says many of these mental hospitals were so horrible they were shut down . where did the patients go ? nowhere . the streets . they became , in many cases , the homeless , he says . they never got treatment . leifman says in 1955 there were more than half a million people in state mental hospitals , and today that number has been reduced 90 percent , and 40,000 to 50,000 people are in mental hospitals . the judge says he 's working to change this . starting in 2008 , many inmates who would otherwise have been brought to the `` forgotten floor '' will instead be sent to a new mental health facility -- the first step on a journey toward long-term treatment , not just punishment . leifman says it 's not the complete answer , but it 's a start . leifman says the best part is that it 's a win-win solution . the patients win , the families are relieved , and the state saves money by simply not cycling these prisoners through again and again . and , for leifman , justice is served . e-mail to a friend .\"\n"
     ]
    }
   ],
   "source": [
    "f = filelists[0]\n",
    "reader = open(f, 'rb')\n",
    "while True:\n",
    "    len_bytes = reader.read(8)\n",
    "    print(\"len_bytes:\", len_bytes)\n",
    "    if not len_bytes:\n",
    "        break # finished reading this file\n",
    "    str_len = struct.unpack('q', len_bytes)[0]\n",
    "    print(\"str_len:\", str_len)\n",
    "    example_str = struct.unpack('%ds' % str_len, reader.read(str_len))[0]\n",
    "    print(\"example_str:\", example_str)\n",
    "    break\n",
    "    #yield example_pb2.Example.FromString(example_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example_str = b\"I love you\"\n",
    "example = example_pb2.Example.FromString(example_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "features {\n",
       "  feature {\n",
       "    key: \"abstract\"\n",
       "    value {\n",
       "      bytes_list {\n",
       "        value: \"<s> mentally ill inmates in miami are housed on the `` forgotten floor \\'\\' </s> <s> judge steven leifman says most are there as a result of `` avoidable felonies \\'\\' </s> <s> while cnn tours facility , patient shouts : `` i am the son of the president \\'\\' </s> <s> leifman says the system is unjust and he \\'s fighting for change . </s>\"\n",
       "      }\n",
       "    }\n",
       "  }\n",
       "  feature {\n",
       "    key: \"article\"\n",
       "    value {\n",
       "      bytes_list {\n",
       "        value: \"editor \\'s note : in our behind the scenes series , cnn correspondents share their experiences in covering news and analyze the stories behind the events . here , soledad o\\'brien takes users inside a jail where many of the inmates are mentally ill . an inmate housed on the `` forgotten floor , \\'\\' where many mentally ill inmates are housed in miami before trial . miami , florida -lrb- cnn -rrb- -- the ninth floor of the miami-dade pretrial detention facility is dubbed the `` forgotten floor . \\'\\' here , inmates with the most severe mental illnesses are incarcerated until they \\'re ready to appear in court . most often , they face drug charges or charges of assaulting an officer -- charges that judge steven leifman says are usually `` avoidable felonies . \\'\\' he says the arrests often result from confrontations with police . mentally ill people often wo n\\'t do what they \\'re told when police arrive on the scene -- confrontation seems to exacerbate their illness and they become more paranoid , delusional , and less likely to follow directions , according to leifman . so , they end up on the ninth floor severely mentally disturbed , but not getting any real help because they \\'re in jail . we toured the jail with leifman . he is well known in miami as an advocate for justice and the mentally ill . even though we were not exactly welcomed with open arms by the guards , we were given permission to shoot videotape and tour the floor . go inside the ` forgotten floor \\' \\'\\' at first , it \\'s hard to determine where the people are . the prisoners are wearing sleeveless robes . imagine cutting holes for arms and feet in a heavy wool sleeping bag -- that \\'s kind of what they look like . they \\'re designed to keep the mentally ill patients from injuring themselves . that \\'s also why they have no shoes , laces or mattresses . leifman says about one-third of all people in miami-dade county jails are mentally ill . so , he says , the sheer volume is overwhelming the system , and the result is what we see on the ninth floor . of course , it is a jail , so it \\'s not supposed to be warm and comforting , but the lights glare , the cells are tiny and it \\'s loud . we see two , sometimes three men -- sometimes in the robes , sometimes naked , lying or sitting in their cells . `` i am the son of the president . you need to get me out of here ! \\'\\' one man shouts at me . he is absolutely serious , convinced that help is on the way -- if only he could reach the white house . leifman tells me that these prisoner-patients will often circulate through the system , occasionally stabilizing in a mental hospital , only to return to jail to face their charges . it \\'s brutally unjust , in his mind , and he has become a strong advocate for changing things in miami . over a meal later , we talk about how things got this way for mental patients . leifman says 200 years ago people were considered `` lunatics \\'\\' and they were locked up in jails even if they had no charges against them . they were just considered unfit to be in society . over the years , he says , there was some public outcry , and the mentally ill were moved out of jails and into hospitals . but leifman says many of these mental hospitals were so horrible they were shut down . where did the patients go ? nowhere . the streets . they became , in many cases , the homeless , he says . they never got treatment . leifman says in 1955 there were more than half a million people in state mental hospitals , and today that number has been reduced 90 percent , and 40,000 to 50,000 people are in mental hospitals . the judge says he \\'s working to change this . starting in 2008 , many inmates who would otherwise have been brought to the `` forgotten floor \\'\\' will instead be sent to a new mental health facility -- the first step on a journey toward long-term treatment , not just punishment . leifman says it \\'s not the complete answer , but it \\'s a start . leifman says the best part is that it \\'s a win-win solution . the patients win , the families are relieved , and the state saves money by simply not cycling these prisoners through again and again . and , for leifman , justice is served . e-mail to a friend .\"\n",
       "      }\n",
       "    }\n",
       "  }\n",
       "}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "’\n"
     ]
    }
   ],
   "source": [
    "print(u'\\u2019')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "”\n"
     ]
    }
   ],
   "source": [
    "print(u'\\u201d')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x08\\x00\\x00\\x00\\x00\\x00\\x00\\x00'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "st = \"I love NLP\"\n",
    "struct.pack('q', 8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
